Extended W3C media extensions for processing dash and CMAF inband events

ABSTRACT

There is included a method and apparatus comprising computer code configured to cause a processor or processors to perform obtaining media data, generating, from the media data, one or more event segments, appending the one or more event segments, to a first event processing buffer, the one or more event segments comprising an event start offset for each of the one or more event segments based on a time at which the each of the one or more event segments is appended to the first event processing buffer, appending the one or more event segments, to a second event processing buffer, the one or more event segments comprising event dispatch information for the each of the one or more event segments, and dispatching the one or more event segments based on the first event processing buffer and the event dispatch information in the second event processing buffer.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority to provisional application U.S.63/176,748, filed on Apr. 19, 2021, the contents of which are herebyexpressly incorporated by reference, in their entirety, into the presentapplication.

FIELD

Embodiments of the present disclosure are directed to the streamingmedia content, and more particularly to streaming media content inaccordance with Moving Picture Experts Group (MPEG) dynamic adaptivestreaming over hypertext transfer protocol (DASH).

BACKGROUND

MPEG DASH provides a standard for streaming media content over IPnetworks. In MPEG DASH, media presentation description (MPD) and inbandevents are used for delivering media timeline related events to aclient. DASH provides a mechanism for media segments to carry inbandevents. Similarly, Common Media Application Format (CMAF) provides amechanism for CMAF chunks to carry inband events. One popular DASH eventis inband MPD validity expiration events. Other events includeapplication events such as SCTE-35 (“Digital Program Insertion CueingMessage for Cable”) events.

Streaming players, including DASH players, use Media Source Extensions(MSE), which allow browsers or user agents to process media segments.However, current MSE specifications do not support parsing andprocessing inband events embedded in DASH or CMAF media segments. Abrowser or user agent, utilizing the current MSE specifications isunable to natively process DASH or CMAF inband events and dispatch theinband events to the respective application.

SUMMARY

The present disclosure addresses one or more technical problems. Thepresent disclosure provides technical solutions that enable a MSE tonatively process DASH and CMAF inband event boxes. The presentdisclosure provides technical solutions allowing a MSE to parse andprocess DASH or CMAF inband events contained in media segments. Thus,browsers or user agents utilizing a MSE based on the present disclosureare able to natively process DASH and CMAF inband event boxes and sendthem to the respective application.

The present disclosure also addresses time synchronization betweeninband events and the respective application. Previous methods includeusing an event's start time in the MSE while processing an event in anapplication. However, such previous methods require maintaining a commontime reference. The present disclosure provides an alternative way fortime synchronization using an offset start time indicating an event'sstart time based on the dispatch time. Using the offset time reducesnetwork overhead, server computational overhead, and applicationcomputational overhead.

Embodiments of the present disclosure provides solutions that enable aMSE to natively process DASH and CMAF inband event boxes contained inmedia segments.

The present disclosure includes a method and apparatus comprising memoryconfigured to store computer program code and a processor or processorsconfigured to access the computer program code and operate as instructedby the computer program code. The computer program code comprises firstobtaining code configure to cause the at least one processor to obtainmedia data, first generating code configured to cause the at least oneprocessor to generate, from the media data, one or more event segments,first appending code configured to cause the at least one processor toappend the one or more event segments, to a first event processingbuffer, the one or more event segments comprising an event start offsetfor each of the one or more event segments based on a time at which theeach of the one or more event segments is appended to the first eventprocessing buffer, second appending code configured to cause the atleast one processor to append the one or more event segments, to asecond event processing buffer, the one or more event segmentscomprising event dispatch information for the each of the one or moreevent segments, and first dispatching code configured to cause the atleast one processor to dispatch the one or more event segments based onthe first event processing buffer and the event dispatch information inthe second event processing buffer.

According to exemplary embodiments, the of the one or more eventsegments to the first event processing buffer causes the each of the oneor more event segments to align respective presentation time andduration of the each of the one or more event segments with respectivepresentation time and duration of at least one associated media samplein a media buffer.

According to exemplary embodiments, the appending of the one or moreevent segments to the second event processing buffer comprisesduplicating the appending of the one or more event segments to the firstevent processing buffer.

According to exemplary embodiments, the appending of the one or moreevent segments to the second event processing buffer is based on theevent dispatch information for the each of the one or more eventsegments.

According to exemplary embodiments, the event dispatch informationcomprises information indicating at least one of event initialization,event appending, event purging, event duration, or event overwrite.

According to exemplary embodiments, the first dispatching code furthercomprises a first determining code configured to cause the at least oneprocessor to determine whether the one or more event segments isincluded in a dispatch event table, and based on determining that theone or more event segments is not included in the dispatch event table,second dispatching code configured to cause the at least one processorto dispatch the one or more event segments to an application.

According to exemplary embodiments, the computer program code furthercomprises first addition code configured to cause the at least oneprocessor to add the one or more event segments to a dispatch eventtable, after dispatching the one or more event segments.

According to exemplary embodiments, the computer program code furthercomprises second generating code configured to cause the at least oneprocessor to generate one or more new event segments, first spittingcode configured to cause the at least one processor to split the one ormore event segments in the first event processing buffer and the secondevent processing buffer based on event duration of the one or more newevent segments, and first overwriting code configured to cause the atleast one processor to overwrite the one or more event segments in thefirst event processing buffer and the second event processing bufferbased on the event duration of the one or more new event segments.

According to exemplary embodiments, the first overwriting code furthercomprises second determining code configured to cause the at least oneprocessor to determine that the one or more event segments in the firstevent processing buffer are not associated with a same media segment,first deleting code configured to cause the at least one processor todelete the one or more event segments from the second event processingbuffer, second deleting code configured to cause the at least oneprocessor to delete the one or more event segments from the first eventprocessing buffer, and third appending code configured to cause the atleast one processor to append the one or more new event segments to thefirst event buffer and the second event processing buffer.

According to exemplary embodiments, the each of the one or more eventsegments is associated with at least one media sample in a media buffer.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, nature, and various advantages of the disclosedsubject matter will be more apparent from the following detaileddescription and the accompanying drawings in which:

FIG. 1 is a simplified illustration of a communication system inaccordance with embodiments.

FIG. 2 is an example illustration of placements of components in astreaming environment in accordance with embodiments.

FIG. 3 is a simplified block diagram of a DASH processing model inaccordance with embodiments.

FIG. 4 is a simplified block diagram of a DASH processing model inaccordance with embodiments.

FIG. 5 is a simplified diagram of media buffer and event buffers inaccordance with embodiments.

FIG. 6 is a simplified diagram of media buffer and event buffers inaccordance with embodiments.

FIG. 7 is a simplified diagram of media buffer and event buffers inaccordance with embodiments.

FIG. 8A is a simplified flowchart illustrating a process for processingDASH and CMAF inband events in accordance with embodiments.

FIG. 8B is a simplified flowchart illustrating a process for processingDASH and CMAF inband events in accordance with embodiments.

FIG. 9 is a simplified flowchart illustrating a process for processingDASH and CMAF inband events in accordance with embodiments.

FIG. 10 is a simplified diagram of a computer system in accordance withembodiments.

DETAILED DESCRIPTION

The proposed features discussed below may be used separately or combinedin any order. Further, the embodiments may be implemented by processingcircuitry (e.g., one or more processors or one or more integratedcircuits). In one example, the one or more processors execute a programthat is stored in a non-transitory computer-readable medium.

FIG. 1 illustrates a simplified block diagram of a communication system100 according to an embodiment of the present disclosure. Thecommunication system 100 may include at least two terminals 102 and 103interconnected via a network 105. For unidirectional transmission ofdata, a first terminal 103 may code video data at a local location fortransmission to the other terminal 102 via the network 105. The secondterminal 102 may receive the coded video data of the other terminal fromthe network 105, decode the coded data and display the recovered videodata. Unidirectional data transmission may be common in media servingapplications and the like.

FIG. 1 illustrates a second pair of terminals 101 and 104 provided tosupport bidirectional transmission of coded video that may occur, forexample, during videoconferencing. For bidirectional transmission ofdata, each terminal 101 and/or 104 may code video data captured at alocal location for transmission to the other terminal via the network105. Each terminal 101 and/or 104 also may receive the coded video datatransmitted by the other terminal, may decode the coded data and maydisplay the recovered video data at a local display device.

In FIG. 1 , the terminals 101, 102, 103 and/or 104 may be illustrated asservers, personal computers and smart phones but the principles of thepresent disclosure are not so limited. Embodiments of the presentdisclosure find application with laptop computers, tablet computers,media players and/or dedicated video conferencing equipment. The network105 represents any number of networks that convey coded video data amongthe terminals 101, 102, 103 and/or 104, including for example wirelineand/or wireless communication networks. The communication network 105may exchange data in circuit-switched and/or packet-switched channels.Representative networks include telecommunications networks, local areanetworks, wide area networks and/or the Internet. For the purposes ofthe present discussion, the architecture and topology of the network 105may be immaterial to the operation of the present disclosure unlessexplained herein below.

FIG. 2 illustrates, as an example, the placement of a video encoder anddecoder in a streaming environment. Embodiments may be applicable toother video enabled applications, including, for example, videoconferencing, digital TV, storing of compressed video on digital mediaincluding CD, DVD, memory stick and the like, and so on.

A streaming system may include a capture subsystem 203, that can includea video source 201, for example a digital camera, creating, for example,an uncompressed video sample stream 213. That sample stream 213 may beemphasized as a high data volume when compared to encoded videobitstreams and can be processed by an encoder 202 coupled to the videosource 201. The encoder 202 can include hardware, software, or acombination thereof to enable or implement aspects of embodiments asdescribed in more detail below. The encoded video bitstream 204, whichmay be emphasized as a lower data volume when compared to the samplestream, can be stored on a streaming server 205 for future use. One ormore streaming clients 212 and 207 can access the streaming server 205to retrieve encoded video bitstream 208 and 206 which may be copies ofthe encoded video bitstream 204. A client 212 can include a videodecoder 211 which decodes the incoming copy of the encoded videobitstream 208 and creates an outgoing video sample stream 210 that canbe rendered on a display 209 or other rendering device. In somestreaming systems, the encoded video bitstreams 204, 206 and 208 can beencoded according to certain video coding standards and/or videocompression standards. Examples of those standards are noted above anddescribed further herein.

A 5G media streaming (5GMS) system may be an assembly of applicationfunctions, application servers, and interfaces from the 5G mediastreaming architecture that support either downlink media streamingservices or uplink media streaming services, or both. A 5GMS ApplicationProvider may include a party that interacts with functions of the 5GMSsystem and supplies a 5GMS Aware Application that interacts withfunctions of the 5GMS system. The 5GMS Aware Application may refer to anapplication in the user equipment (UE), provided by the 5GMS ApplicationProvider, that contains the service logic of the 5GMS applicationservice, and interacts with other 5GMS Client and Network functions viathe interfaces and application programming interfaces (APIs) defined inthe 5GMS architecture. A 5GMS Client may refer to a UE function that iseither a 5GMS downlink (5GMSd) Client or a 5GMS uplink (5GMSu) Client,or both.

The 5GMSd Client may refer to a UE function that includes at least a 5Gmedia streaming player and a media session handler for downlinkstreaming and that may be accessed through well-defined interfaces/APIs.The 5GMSu Client may refer to an originator of a 5GMSu service that maybe accessed through well-defined interfaces/APIs. A 5GMSu media streamermay refer to a UE function that enables uplink delivery of streamingmedia content to an Application Server (AS) function of the 5GMSApplication Provider, and which interacts with both the 5GMSu AwareApplication for media capture and subsequent streaming, and the MediaSession Handler for media session control.

A dynamic policy may refer to a dynamic policy and charging control(PCC) rule for an uplink or downlink application flow during a mediasession. An egest session may refer to an uplink media streaming sessionfrom the 5GMS AS towards the 5GMSu Application Provider. An ingestsession may refer to a session to upload the media content to a 5GMSdAS. A policy template may refer to a collection of (semi-static) Policyor Control Function (PCF)/Network Exposure Function (NEF) API parameterswhich are specific to the 5GMS Application Provider and also theresulting PCC rule. A policy template ID may identify the desired policytemplate, which is used by the 5GMSd Application Function (AF) to selectthe appropriate PCF/NEF API towards the 5G system so that the PCF cancompile the desired PCC rule. The Media Player Entry may refer to adocument or a pointer to a document that defines a media presentation(e.g., a media presentation description (MPD) for DASH or a uniformresource locator (URL) to a video clip file). A Media Streamer Entry mayrefer to a pointer (e.g., in the form of a URL) that defines an entrypoint of an uplink media streaming session. A presentation entry mayrefer to a document or a pointer to a document that defines anapplication presentation, such as an HTML5 document.

A Provisioning Session may refer to a data structure supplied at aninterface (M1d) by a 5GMSd Application provider that configures the5GMSd features relevant to a set of 5GMSd Aware Applications. A 5GMSdMedia Player may refer to a UE function that enables playback andrendering of a media presentation based on a media play entry andexposing some basic controls such as play, pause, seek, stop, to the5GMSd Aware Application. Server Access Information may refer to a set ofparameters and addresses (including 5GMSd AF and 5GMSd AS addresses)which are needed to activate the reception of a streaming session. AService and Content Discovery may refer to functionality and proceduresprovided by a 5GMSd Application Provider to a 5GMS Aware Applicationthat enables the end user to discover the available streaming serviceand content offerings and select a specific service or content item foraccess. A Service Announcement may refer to procedures conducted betweenthe 5GMS Aware Application and the 5GMS Application Provider such thatthe 5GMS Aware Application is able to obtain 5GMS Service AccessInformation, either directly or in the form of a reference to thatinformation.

A third party player may refer to a part of an application that usesAPIs to exercise selected 5GMSd functions to play back media content. Athird party uplink streamer may refer to a part of an application thatuses APIs to exercise selected 5GMSu functions to capture and streammedia content.

FIG. 3 shows a sample DASH processing model 300, such as of a sampleclient architecture for processing DASH and CMAF events. In the DASHprocessing model 300, a client's request of media segments may be basedon described addresses in a manifest 303. The manifest 303 alsodescribes metadata tracks from which a client may access segments ofmetadata tracks, parse them, and send them to an application 301.

The manifest 303 includes MPD events or inband events, and an inbandevent and ‘moof’ parser 306 may parse MPD event segments or inband eventsegments and append the event segments to an event and metadata buffer330. The inband event and ‘moof’ parser 306 may also fetch and appendthe media segments to a media buffer 340. The event and metadata buffer330 may send event and metadata information to an event and metadatasynchronizer and dispatcher 335. The event and metadata synchronizer anddispatcher 335 may dispatch specific events to DASH players control,selection, and heuristic logic 302 and application related events andmetadata tracks to application 301.

According to some embodiments, a MSE may include the media buffer 340and a media decoder 345. MSE 320 is a logical buffer(s) of mediasegments, where the media segments may be tracked and ordered based onthe media segments' presentation time. Each media segment may be addedor appended to the media buffer 340 based on the media segments'timestamp offset, and the timestamp offset may be used to order themedia segments in the media buffer 340.

As long as media segments exist in the media buffer 340, the event andmetadata buffer 330 maintains corresponding event segments and metadata.According to FIG. 3 , the MSE 320 includes only the media buffer 340 andthe media decoder 345. The event and metadata buffer 330 and event andmetadata synchronizer and dispatcher 335 are not native to the MSE 320,inhibiting the MSE 320 from natively processing events and sending themto the application.

FIG. 4 shows a sample DASH processing model 400, such as of a such as ofa sample client architecture for processing DASH and CMAF events. In theDASH processing model 400, a client request of media segments may bebased on described addresses in a manifest 403. The manifest 403 alsodescribes metadata tracks from which a client may access segments ofmetadata tracks, parse them, and send them to an application 401.

The manifest 403 includes MPD events or inband events, and an inbandevent and ‘moof’ parser 406 may parse the MPD event segments or inbandevent segments and append the event segments to an event purging buffer430. Based on the media data or media stream and information in themanifest 403, the inband event and ‘moof’ parser 406 may fetch andappend the media segments to a media buffer 440. The event purgingbuffer 330 may send event and metadata information to an eventsynchronizer and dispatcher 435. The event synchronizer and dispatcher435 may dispatch specific events to DASH players control, selection, andheuristic logic 402 and application related events and metadata tracksto application 401.

According to exemplary embodiments, in FIG. 4 , the MSE is extended toinclude the media buffer 340, media decoder 345, event purging buffer430, event synchronizer and dispatcher 435, and an already-dispatchedevent table 450. The already-dispatched event table 450 may record theevents that are already dispatched to the application 401. The extensionof event purging buffer 430 and event synchronizer and dispatcher 435,enables the MSE to process inband events natively and the creation ofalready-dispatched event table 450 enables recording and tracking ofalready dispatched event messages.

In exemplary embodiments, the MSE 420 or its components may dispatchevents based on an event segment's event specific offset. An exampleincludes event start time offset. The MSE 420 or its components maydetermine event start time offset for an event segment, wherein thestart time offset is determined with reference to event presentationtime, and use the event start time offset to order and dispatch eventsin the event purging buffer 430 or event synchronizer and dispatcher435. In exemplary embodiments, the event start time offset and timestampoffset may be equivalent, with the event start time offset referring toevent segments in the event purging buffer 430 or the event synchronizerand dispatcher 435 and the time stamp offset referring to media segmentsin the media buffer 440.

In exemplary embodiments, the MSE 420 may handle event segment and mediasegment purging and overwriting for associated events and media. The MSE420 may retain an event in the event purging buffer 430 or the eventsynchronizer and dispatcher 435 if the media segment associated with theevent is retained in the media buffer 440. The MSE 420 may manage timingand duration of the event segments based on the timing and duration ofthe associated media segments.

In exemplary embodiments, the application 401 may set the timestampoffset or the event time stamp offset. The application 401 may also setthe scheme URI/value and the dispatch mode for the scheme URI/value. TheMSE 420 may dispatch events based on scheme URI/value, the event ID, theevent association ID that indicates the media segment associated withthe event segment, the event segment's presentation time, the event'sstart time offset, and event duration.

FIG. 5 shows an example media and event buffer set 500 processing mediasegments and event segments. Each media stream 510 may comprise of mediasegments (S0, S1, S2, S3) and event segments (E0, E1, E2). The mediasegments may be appended to a media source buffer or media buffer 520.

An inband event and ‘moof’ parser 406 may parse the media stream and maygenerate media segments and event segments. Each media segment may beappended to the media source buffer or media buffer 520 based on atimestamp offset. Each media segment appended to the media source bufferor media buffer 520 may have a unique event association ID. Thegenerated event segments (E0, E1, E2) may be associated with a specificmedia segment (S0, S1, S2, S3) using the respective media segment'sevent association ID. In exemplary embodiments, multiple event segmentsmay be associated with a media segment.

In exemplary embodiments, event segments (E0, E1, E2) may be appended tothe event purge buffer 530 by aligning the event segments' event starttime and event duration in the event purge buffer 530 with the starttime and duration of the associated media segment (S0, S1, S2, S3) inthe media buffer 520. As an example, one or more event segments'presentation time or event start time offset and duration in the eventpurge buffer 530 may be aligned with the presentation time and durationof the associated media segment in the media buffer 620. In someexemplary embodiments, if more than one event segment is associated witha same media segment, then each event segment's event start time offsetmay be adjusted based on the associated media segment's presentationtime.

In some exemplary embodiments, event segments may be appended to theevent dispatch buffer 540 by duplicating the respective event segments'event start times and durations in the event purge buffer 530. In someembodiments, order, presentation time, and duration of event segmentsmay be equivalent in both the event purge buffer 530 and the eventdispatch buffer 540. As an example, if the application has set thescheme as “on_receive” event segments' presentation time or event starttime offset and duration in the event dispatch buffer 540 may be alignedwith the presentation time and duration of the associated media segmenti.e., the event dispatch buffer 540 may be a duplication of the eventpurge buffer 530.

FIG. 6 shows an example media and event buffer set processing mediasegments and event segments. Each media stream 610 may comprise of mediasegments (S0, S1, S2, S3) and event segments (E0, E1, E2). The mediasegments may be appended to a media source buffer or media buffer 620.

An inband event and ‘moof’ parser 406 may parse the media stream and maygenerate media segments and event segments. Each media segment may beappended to the media source buffer or media buffer 620 based on atimestamp offset. Each media segment appended to the media source bufferor media buffer 620 may have a unique event association ID. Thegenerated event segments (E0, E1, E2) may be associated with a specificmedia segment (S0, S1, S2, S3) using the respective media segment'sevent association ID. In exemplary embodiments, multiple event segmentsmay be associated with a media segment.

In exemplary embodiments, event segments (E0, E1, E2) may be appended tothe event purge buffer 630 by aligning the event segments' event starttime and event duration in the event purge buffer 630 with the starttime and duration of the associated media segment (S0, S1, S2, S3) inthe media buffer 620. As an example, one or more event segments'presentation time or event start time offset and duration in the eventpurge buffer 630 may be aligned with the presentation time and durationof the associated media segment in the media buffer 620. In someexemplary embodiments, if more than one event segment is associated withthe same media segment, then each event segment's event start timeoffset should be adjusted based on the associated media segment'spresentation time.

In exemplary embodiments, event segments may be appended to the eventdispatch buffer 640 by adding event segments such that the eventsegments' event start time offset is the event's presentation time inthe event dispatch buffer 640 and the event segment's duration is theevent's duration in the event dispatch buffer 640. As an example, if theapplication has set the scheme as “on_start” event segments'presentation time and duration in the event dispatch buffer 640 mayreflect the event segments' event start time offset and duration. Insome embodiments, the event dispatch buffer 640 may reflect the eventsegment's event start time and duration, i.e., the event segment may beappended such that the event segment is in the event dispatch buffer 640from the event segment's start until the event segment's end. Thus, insome embodiments, the event purge buffer 630 and the event dispatchbuffer 640 may not be equivalent.

FIG. 7 shows in example media and event buffer set 700 for processingmedia segments and event segments. Each media stream 710 may comprise ofmedia segments (S0, S1, S2, S3) and event segments (E0, E1, E2). Themedia segments may be appended to a media source buffer or media buffer720.

In exemplary embodiments, event segments (E0, E1, E2) in an event purgebuffer 730 and an event dispatch buffer 740 may be removed, purged, oroverwritten. In some embodiments, event segments may be partiallyremoved, purged, or overwritten. To remove, purge, or overwrite eventsegments, the event segments may be removed from the event purge buffer730 and the event dispatch buffer 740. As an example, once a mediasegment is overwritten in the media buffer 720, the corresponding eventsegments in the event purge buffer 730 and event dispatch buffer 740that associated with the overwritten media segments may be overwrittenas well.

In some embodiments, to remove or overwrite event segments partially,event segments may be split into smaller sections in the event purgebuffer 730 and the event dispatch buffer 740, and only the sections ofthe event segments that overlap with new event segments may be removed,purged, or overwritten and the new event segment sections added instead.As an example, if new event segments are smaller than the current eventsegments in the event purge buffer 730 or the event dispatch buffer 740,the current event segment(s) in the event purge buffer 730 may be splitinto smaller sections to match the new event segments. The sections ofthe event segments in the event purge buffer 730 that overlap with thenew smaller event segments may be removed, purged, or overwritten.

In some embodiments, only the event segments in the event purge buffer730 that overlap the new event segments may be removed, purged, oroverwritten. In some embodiments only the event segments in the eventdispatch buffer 740 that overlap the new event segments may be removed,purged, or overwritten. The event purge buffer 730 may be overwrittenand then the event dispatch buffer 740 may be overwritten such that theevent purge buffer 730 and the event dispatch buffer 740 may beequivalent. As an example, if the application has set the scheme as“on_receive” the event segments or event segment sections that overlapwith the new event segments in both the event purge buffer 730 and theevent dispatch buffer 740 may be overwritten.

In some exemplary embodiments, removal, purge, or overwriting of eventsegments may require determining that there is no other event segment inthe event purge buffer 730 that is associated with the same mediasegment. In some exemplary embodiments, if there is no other eventsegment in the event purge buffer 730 associated with the same mediasegment, then the corresponding event in the event dispatch buffer 740may be removed, purged, or overwritten. As an example, if more than oneevent segment overlaps the new event segment in the event purge buffer730, overwriting the overlapping event segment may require that all theevent segments associated with the same media segment are dispatchedbefore overwriting the overlapping event segment in the event purgebuffer 730. In some embodiments, if there are no event segmentsassociated with the same media segment in the event purge buffer 730,the overlapping event segment may be overwritten in the event purgebuffer 730. As an example, if the application has set the scheme to“on_start” overwriting an overlapping event segment in the event purgebuffer 730 may require that all the event segments associated with thesame media segment are dispatched before overwriting the overlappingevent segment in the event purge buffer 730. In some embodiments, oncethe event segment is removed, purged, or overwritten in the event purgebuffer 730, the corresponding event segment may be removed, purged, oroverwritten in the event dispatch buffer 740. As an example, if theapplication has set the scheme to “on_start” the overlapping segment tobe overwritten in the event dispatch buffer 740 may be overwritten afterthe overlapping event segment(s) in the event purge buffer 730 areoverwritten.

FIG. 8A shows an exemplary flowchart illustrating a process 800 withrespect to processing DASH and CMAF inband events wherein according toexemplary embodiments event segments may be appended to an event purgebuffer and an event dispatch buffer. At 810, media data is obtained. Themedia data may be any type of media data. As an example, the media datamay be audio data, video data, document, or any combination of thereof.The obtained media data may also contain event data.

At 815, event segments may be generated from the obtained media data. Insome embodiments, an MSE or a parser component of the MSE may generateevent segments from the obtained media data. Event segments may includeMPDs and inband events to deliver media timeline related events to aclient or an application. As an example, the inband event and ‘moof’parser (306 or 406) may generate event segments from the obtained mediadata. In some embodiments, the parser may generate both media segmentsand event segments corresponding to the media segments. As an example,the inband event and ‘moof’ parser (306 or 406) may generate mediasegments and associated event segments from the media data. The eventsegments may be associated with event segments using an eventassociation ID, wherein the event association ID may be a unique IDassociated with a media segment. In some embodiments, multiple eventsegments may be associated with the same media segment. Event segmentsassociated with the same media segment may have the event associationID.

At 820, the event start time offset of an event segment is determined.In some embodiments, the MSE or its components may determine the eventstart time offset for each event segment to be appended to the eventpurge buffer or the event dispatch buffer. In some embodiments, theevent start time offset for an event segment may be based on the time atwhich the event segment is appended to the event purging buffer. In someembodiments, the event start time offset for an event segment may bebased on the time at which the event segment is appended to the eventdispatch buffer. In some embodiments, an event segment's event starttime offset may be based on the event segment's or media segment'spresentation time in the MSE API. The event segment's start time offsetindicates the start time of the event from the moment of dispatch. Thus,when an event segment is sent to an application, the application candetermine the relative start of the event from the moment of dispatchinstead of comparing common time reference. Using an event segment'sevent start time offset based on the time in the MSE API eliminates theneed for an application to compare common time reference whendetermining relative start times for event segments, increasingapplication processing efficiency and reducing bandwidth requirements.

At 825, event segments may be appended to an event purge buffer based onthe event segment's event start time offset. The event purge buffer maybe used to track event segments' order, their location, and dispatchtiming. In some embodiments, appending an event segment to the purgingbuffer may include aligning the presentation time and duration of theevent segment with the respective presentation time and duration of theassociated media segment. As an example, event segments associated witha media segment may be placed in the event purge buffer such that theevent segments have the same presentation time and duration as that ofthe media segments associated with each event segment.

At 830, the event segments may be appended to the an event dispatchbuffer based on the event segment's dispatch information. In someembodiments dispatch information may include information indicating atleast one of event initialization, event appending, event purging, eventduration, event overwrite, or attributes set by an application. Someattributes set by the application or based on attributes set by theapplication may include scheme URI/value, id, event segment's eventstart time offset, event segment's start presentation delta and eventsegment duration.

In some embodiments, appending the event segment to the event dispatchbuffer may include replicating or duplicating the appending of the eventsegment to the event purge buffer. Thus, the order, presentation time,and duration of event segments may be the same in the event purge bufferand the event dispatch buffer. As an example, if the application has setthe scheme as “on_receive” one or more event segments' presentation timeor event start time offset and duration in the event dispatch buffer maybe aligned with the presentation time and duration of the associatedmedia segment i.e., the event dispatch buffer may be a duplication ofthe event purge buffer.

In some embodiments, appending the event segment to the event dispatchbuffer may be achieved by adding event segments in a way that the eventsegment's event start time offset is the event's presentation time inthe event dispatch buffer and the event segment's duration is theevent's duration in the event dispatch buffer. As an example, if theapplication has set the scheme as “on_start” the event segment'spresentation time and duration in the event dispatch buffer may reflectthe event segment's event start time offset and duration. In someembodiments, the event dispatch buffer may reflect the event segment'sevent start time and duration, i.e., the event segment may be appendedsuch that the event segment is in the buffer from the event segment'sstart until the event segment's end. Thus, in some embodiments, theevent purge buffer and the event dispatch buffer may not be equivalent.

At 835, event segments may be dispatched to the application based ondispatch information for respective event segments. Dispatch informationmay include information about event segment initialization, eventassociation ID, event start time offset, event segment duration, theapplication scheme URI/value, and event purging, event appending, eventoverwrite attributes.

FIG. 8B shows an exemplary flowchart illustrating a process 850 withrespect to processing DASH and CMAF inband events wherein according toexemplary embodiments event segments may be dispatched from an eventpurge buffer and an event dispatch buffer.

At 860, event segments in the event dispatch buffer for the specificpresentation time are identified. In some embodiments, an MSE or itscomponents may identify the event segments in the event dispatch bufferfor the specific presentation time. In some embodiments, thepresentation time may be based on the MSE API event presentation time.In other embodiments, the presentation time may be based on thepresentation time for media segments. Multiple event segments may beassociated with an event association ID, i.e., multiple event segmentsmay be associated with a same media segment.

At 865, whether an event segment identified in the event dispatch bufferfor the specific presentation time is included in a dispatch event tableis determined. In some embodiments, the MSE or its components maydetermine whether an event segment is included in an dispatch eventtable. The dispatch event table may keep a record of event segments orevent messages that have already been dispatched to an application.Thus, if an event segment is included in the dispatch event table, thenthat event segment may already be dispatched to an application. If anevent segment in the event dispatch buffer at the presentation time isnot included in the dispatch event table, the event segment maydispatched to the application at 870 because that event segment islikely not dispatched to the application yet. As an example, if an eventID of an event segment in the event dispatch buffer at the presentationtime is not present in the dispatch event table, then the event segmentis dispatched to an application. As another example, if the event ID ofan event segment in the event dispatch buffer at the presentation timeis included in the dispatch event table, then that event segment mayalready be dispatched to the application.

At 875, event segments that are dispatched to an application may beadded to the dispatch event table. In some embodiments, event segmentsthat are dispatched and not included in the dispatch event table areadded to the dispatch event table. As an example, if an event segmentwas dispatched to an application then the event ID of that event segmentmay be added to the dispatch event table to maintain a records that thatevent segment was dispatched to the application.

FIG. 9 shows an exemplary flowchart illustrating a process 900 withrespect to processing DASH and CMAF inband events wherein according toexemplary embodiments event segments may be removed, purged, oroverwritten from an event purge buffer and an event dispatch buffer. At910, new event segments from new media data are generated. In someembodiments, an MSE or its components may generate one or more new eventsegments from new media data. In some exemplary embodiments, new eventsegments may be generated for new media segments being appended to themedia buffer. As an example, when new media segments are parsed andappended to the media buffer, new event segments corresponding to thenew media segments may be appended to the event purge buffer and eventdispatch buffer.

At 915, event segments overlapping with new event segments in the eventpurge buffer may be spilt to match the event duration, event start timeoffset, or presentation time of the new event segments. If the newsegments are smaller than the event segments in the event purge buffer,the overlapping event segments in the event purge buffer may be split tomatch the event duration of the new event segments. As an example, ifthe new event segments that overlap with event segments in the eventpurge buffer have shorter event duration, the overlapping event segmentsin the event purge buffer may be split into two or more sectionscorresponding to the shorter event duration of the new event segments.In some embodiments, the event segment sections in the event purgebuffer that overlap are removed, purged, or overwritten based on theduration of the new event segments.

At 920, event segments overlapping with new event segments in the eventdispatch buffer may be spilt to match the event duration, event starttime offset, or presentation time of the new event segments. If the newsegments are smaller or shorter than the event segments in the eventdispatch buffer, the overlapping event segments in the event dispatchbuffer may be split to match the event duration, event start timeoffset, or presentation time of the new event segments. As an example,if the new event segments that overlap with event segments in the eventdispatch buffer have shorter event duration, the overlapping eventsegments in the event dispatch buffer may be split into two or moresections corresponding to the shorter event duration of the new eventsegments. In some embodiments, the event segment sections in the eventdispatch buffer that overlap are removed, purged, or overwritten basedon the duration of the new event segments.

At 925, whether event segments from the event purge buffer and the eventdispatch buffer may be removed, purged, or overwritten is determined.Whether event segments from the event purge buffer or the event dispatchbuffer may be removed, purged, or overwritten depends on whether allevent segments associated with a media segment being overwritten aredispatched. If event segments from the event purge buffer that overlapwith the new event segments are not associated with the same mediasegment, at 930, the event segments from the event purge buffer thatoverlap with the new event segments may be deleted from the eventdispatch buffer. At 935, event segments from the event purge buffer thatoverlap with the new event segments may be deleted from the event purgebuffer. As an example, if the application has set the scheme as“on_start” and if event segments from the event purge buffer thatoverlap with the new event segments are not associated with the samemedia segment, then the event segments that overlap with the new eventsegments may be deleted from the event purge buffer and may also bedeleted from the event dispatch buffer because there are other eventsassociated with the same media segment have likely been dispatchedalready.

In some embodiments, the step of determining whether overlapping eventsegments in the event purge buffer are not associated with the samemedia segment may be optionally performed. As an example, if anapplication has set the scheme to “on_receive” then the step ofdetermining whether overlapping event segments in the event purge bufferare not associated with the same media segment may not be performed.

At 930, the event segments that overlap with the new event segments maybe deleted from the event purge buffer. At 935, the event segments thatoverlap with the new event segments may be deleted from the eventdispatch buffer. At 940, the new event segments may be appended to theevent purge buffer and the event dispatch buffer in accordance with theembodiments in the present disclosure.

Although FIGS. 8A-8B and 9 shows example blocks of the processes 800,850, and 900, in embodiments, the processes 800, 850, and 900 mayinclude additional blocks, fewer blocks, different blocks, ordifferently arranged blocks than those depicted in FIGS. 8A-8B and 9 .In embodiments, any blocks of processes 800, 850, and 900 may becombined or arranged in any amount or order, as desired. In embodiments,two or more of the blocks of the processes 800, 850, and 900 may beperformed in parallel.

The techniques described above, can be implemented as computer softwareusing computer-readable instructions and physically stored in one ormore computer-readable media or by a specifically configured one or morehardware processors. For example, FIG. 10 shows a computer system 1000suitable for implementing various embodiments.

The computer software can be coded using any suitable machine code orcomputer language, that may be subject to assembly, compilation,linking, or like mechanisms to create code comprising instructions thatcan be executed directly, or through interpretation, micro-codeexecution, and the like, by computer central processing units (CPUs),Graphics Processing Units (GPUs), and the like.

The instructions can be executed on various types of computers orcomponents thereof, including, for example, personal computers, tabletcomputers, servers, smartphones, gaming devices, internet of thingsdevices, and the like.

The components shown in FIG. 10 for computer system 1000 are exemplaryin nature and are not intended to suggest any limitation as to the scopeof use or functionality of the computer software implementingembodiments of the present disclosure. Neither should the configurationof components be interpreted as having any dependency or requirementrelating to any one or combination of components illustrated in theexemplary embodiment of a computer system 1000.

Computer system 1000 may include certain human interface input devices.Such a human interface input device may be responsive to input by one ormore human users through, for example, tactile input (such as:keystrokes, swipes, data glove movements), audio input (such as: voice,clapping), visual input (such as: gestures), olfactory input. The humaninterface devices can also be used to capture certain media notnecessarily directly related to conscious input by a human, such asaudio (such as: speech, music, ambient sound), images (such as: scannedimages, photographic images obtain from a still image camera), video(such as two-dimensional video, three-dimensional video includingstereoscopic video).

Input human interface devices may include one or more of (only one ofeach depicted): keyboard 1001, mouse 1002, trackpad 1003, touch screen1010, joystick 1005, microphone 1006, scanner 1008, camera 1007.

Computer system 1000 may also include certain human interface outputdevices. Such human interface output devices may be stimulating thesenses of one or more human users through, for example, tactile output,sound, light, and smell/taste. Such human interface output devices mayinclude tactile output devices (for example tactile feedback by thetouch screen 1010, or joystick 1005, but there can also be tactilefeedback devices that do not serve as input devices), audio outputdevices (such as: speakers 1009, headphones), visual output devices(such as screens 1010 to include CRT screens, LCD screens, plasmascreens, OLED screens, each with or without touch-screen inputcapability, each with or without tactile feedback capability—some ofwhich may be capable to output two dimensional visual output or morethan three dimensional output through means such as stereographicoutput; virtual-reality glasses, holographic displays and smoke tanks),and printers.

Computer system 1000 can also include human accessible storage devicesand their associated media such as optical media including CD/DVD ROM/RW1020 with CD/DVD 1011 or the like media, thumb-drive 1022, removablehard drive or solid state drive 1023, legacy magnetic media such as tapeand floppy disc, specialized ROM/ASIC/PLD based devices such as securitydongles, and the like.

Those skilled in the art should also understand that term “computerreadable media” as used in connection with the presently disclosedsubject matter does not encompass transmission media, carrier waves, orother transitory signals.

Computer system 1000 can also include interface 1099 to one or morecommunication networks 1098. Networks 1098 can for example be wireless,wireline, optical. Networks 1098 can further be local, wide-area,metropolitan, vehicular and industrial, real-time, delay-tolerant, andso on. Examples of networks 1098 include local area networks such asEthernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G,LTE and the like, TV wireline or wireless wide area digital networks toinclude cable TV, satellite TV, and terrestrial broadcast TV, vehicularand industrial to include CANBus, and so forth. Certain networks 1098commonly require external network interface adapters that attached tocertain general-purpose data ports or peripheral buses (1050 and 1051)(such as, for example USB ports of the computer system 1000; others arecommonly integrated into the core of the computer system 1000 byattachment to a system bus as described below (for example Ethernetinterface into a PC computer system or cellular network interface into asmartphone computer system). Using any of these networks 1098, computersystem 1000 can communicate with other entities. Such communication canbe uni-directional, receive only (for example, broadcast TV),uni-directional send-only (for example CANbusto certain CANbus devices),or bi-directional, for example to other computer systems using local orwide area digital networks. Certain protocols and protocol stacks can beused on each of those networks and network interfaces as describedabove.

Aforementioned human interface devices, human-accessible storagedevices, and network interfaces can be attached to a core 1040 of thecomputer system 1000.

The core 1040 can include one or more Central Processing Units (CPU)1041, Graphics Processing Units (GPU) 1042, a graphics adapter 1017,specialized programmable processing units in the form of FieldProgrammable Gate Areas (FPGA) 1043, hardware accelerators for certaintasks 1044, and so forth. These devices, along with Read-only memory(ROM) 1045, Random-access memory 1046, internal mass storage such asinternal non-user accessible hard drives, SSDs, and the like 1047, maybe connected through a system bus 1048. In some computer systems, thesystem bus 1048 can be accessible in the form of one or more physicalplugs to enable extensions by additional CPUs, GPU, and the like. Theperipheral devices can be attached either directly to the core's systembus 1048, or through a peripheral bus 1051. Architectures for aperipheral bus include PCI, USB, and the like.

CPUs 1041, GPUs 1042, FPGAs 1043, and accelerators 1044 can executecertain instructions that, in combination, can make up theaforementioned computer code. That computer code can be stored in ROM1045 or RAM 1046. Transitional data can be also be stored in RAM 1046,whereas permanent data can be stored for example, in the internal massstorage 1047. Fast storage and retrieval to any of the memory devicescan be enabled through the use of cache memory, that can be closelyassociated with one or more CPU 1041, GPU 1042, mass storage 1047, ROM1045, RAM 1046, and the like.

The computer readable media can have computer code thereon forperforming various computer-implemented operations. The media andcomputer code can be those specially designed and constructed for thepurposes of the present disclosure, or they can be of the kind wellknown and available to those having skill in the computer software arts.

As an example and not by way of limitation, the computer system 1000having the illustrated architecture, and specifically the core 1040 canprovide functionality as a result of processor(s) (including CPUs, GPUs,FPGA, accelerators, and the like) executing software embodied in one ormore tangible, computer-readable media. Such computer-readable media canbe media associated with user-accessible mass storage as introducedabove, as well as certain storage of the core 1040 that are ofnon-transitory nature, such as core-internal mass storage 1047 or ROM1045. The software implementing various embodiments of the presentdisclosure can be stored in such devices and executed by core 1040. Acomputer-readable medium can include one or more memory devices orchips, according to particular needs. The software can cause the core1040 and specifically the processors therein (including CPU, GPU, FPGA,and the like) to execute particular processes or particular parts ofparticular processes described herein, including defining datastructures stored in RAM 1046 and modifying such data structuresaccording to the processes defined by the software. In addition or as analternative, the computer system can provide functionality as a resultof logic hardwired or otherwise embodied in a circuit (for example:accelerator 1044), which can operate in place of or together withsoftware to execute particular processes or particular parts ofparticular processes described herein. Reference to software canencompass logic, and vice versa, where appropriate. Reference to acomputer-readable media can encompass a circuit (such as an integratedcircuit (IC)) storing software for execution, a circuit embodying logicfor execution, or both, where appropriate. The present disclosureencompasses any suitable combination of hardware and software.

While this disclosure has described several exemplary embodiments, thereare alterations, permutations, and various substitute equivalents, whichfall within the scope of the disclosure. It will thus be appreciatedthat those skilled in the art will be able to devise numerous systemsand methods which, although not explicitly shown or described herein,embody the principles of the disclosure and are thus within the spiritand scope thereof

What is claimed is:
 1. A method of processing events in a media stream,the method comprising: obtaining media data; generating, from the mediadata, one or more event segments; appending the one or more eventsegments to a first event processing buffer, the one or more eventsegments comprising an event start offset for each of the one or moreevent segments based on a time at which the each of the one or moreevent segments is appended to the first event processing buffer;appending the one or more event segments to a second event processingbuffer, the one or more event segments comprising event dispatchinformation for the each of the one or more event segments; anddispatching the one or more event segments based on the first eventprocessing buffer and the event dispatch information in the second eventprocessing buffer; generating one or more new event segments; splittingthe one or more event segments in the first event processing buffer andthe second event processing buffer based on event duration of one ormore new event segments; and overwriting the one or more event segmentsin the first event processing buffer and the second event processingbuffer based on the event duration of the one or more new eventsegments.
 2. The method according to claim 1, wherein the appending ofthe one or more event segments to the first event processing buffercauses the each of the one or more event segments to align respectivepresentation time and duration of the each of the one or more eventsegments with respective presentation time and duration of at least oneassociated media sample in a media buffer.
 3. The method according toclaim 1, wherein the event dispatch information comprises informationindicating at least one of event initialization, event appending, eventpurging, event duration, or event overwrite.
 4. The method according toclaim 1, wherein the appending of the one or more event segments to thesecond event processing buffer comprises duplicating the appending ofthe one or more event segments to the first event processing buffer. 5.The method according to claim 1, wherein the appending of the one ormore event segments to the second event processing buffer is based onthe event dispatch information for the each of the one or more eventsegments.
 6. The method according to claim 1, wherein the dispatching ofthe one or more event segments comprises: determining whether the one ormore event segments is included in a dispatch event table; and based ondetermining that the one or more event segments is not included in thedispatch event table, dispatching the one or more event segments to anapplication.
 7. The method according to claim 1, further comprising:after the dispatching of the one or more event segments, adding the oneor more event segments to a dispatch event table.
 8. The methodaccording to claim 1, wherein the overwriting of the one or more eventsegments comprises: determining that the one or more event segments inthe first event processing buffer are not associated with a same mediasegment; deleting the one or more event segments from the second eventprocessing buffer; deleting the one or more event segments from thefirst event processing buffer; and appending the one or more new eventsegments to the first event processing buffer and the second eventprocessing buffer.
 9. The method according to claim 1, wherein the eachof the one or more event segments is associated with at least one mediasample in a media buffer.
 10. An apparatus for processing events in amedia stream, the apparatus comprising: at least one memory configuredto store computer program code; at least one processor configured toaccess the computer program code and operate as instructed by thecomputer program code, the computer program code including: firstobtaining code configured to cause the at least one processor to obtainmedia data; first generating code configured to cause the at least oneprocessor to generate, from the media data, one or more event segments;first appending code configured to cause the at least one processor toappend the one or more event segments to a first event processingbuffer, the one or more event segments comprising an event start offsetfor each of the one or more event segments based on a time at which theeach of the one or more event segments is appended to the first eventprocessing buffer; second appending code configured to cause the atleast one processor to append the one or more event segments to a secondevent processing buffer, the one or more event segments comprising eventdispatch information for the each of the one or more event segments; andfirst dispatching code configured to cause the at least one processor todispatch the one or more event segments based on the first eventprocessing buffer and the event dispatch information in the second eventprocessing buffer; second generating code configured to cause the atleast one processor to generate one or more new event segments; firstsplitting code configured to cause the at least one processor to splitthe one or more event segments in the first event processing buffer andthe second event processing buffer based on event duration of the one ormore new event segments; and first overwriting code configured to causethe at least one processor to overwrite the one or more event segmentsin the first event processing buffer and the second event processingbuffer based on the event duration of the one or more new eventsegments.
 11. The apparatus according to claim 10, wherein the appendingof the one or more event segments to the first event processing buffercauses the each of the one or more event segments to align respectivepresentation time and duration of the each of the one or more eventsegments with respective presentation time and duration of at least oneassociated media sample in a media buffer.
 12. The apparatus accordingto claim 10, wherein the event dispatch information comprisesinformation indicating at least one of event initialization, eventappending, event purging, event duration, or event overwrite.
 13. Theapparatus according to claim 10, wherein the appending of the one ormore event segments to the second event processing buffer comprisesduplicating the appending of the one or more event segments to the firstevent processing buffer.
 14. The apparatus according to claim 10,wherein the appending of the one or more event segments to the secondevent processing buffer is based on the event dispatch information forthe each of the one or more event segments.
 15. The apparatus accordingto claim 10, wherein the first dispatching code further comprises: firstdetermining code configured to cause the at least one processor todetermine whether the one or more event segments is included in adispatch event table; and second dispatching code configured to causethe at least one processor to, based on determining that the one or moreevent segments is not included in the dispatch event table, dispatch theone or more event segments to an application.
 16. The apparatusaccording to claim 10, further comprising: first addition codeconfigured to cause the at least one processor to add the one or moreevent segments to a dispatch event table, after dispatching the one ormore event segments.
 17. The apparatus according to claim 10, whereinthe first overwriting code further comprises: second determining codeconfigured to cause the at least one processor to determine that the oneor more event segments in the first event processing buffer are notassociated with a same media segment; first deleting code configured tocause the at least one processor to delete the one or more eventsegments from the second event processing buffer; second deleting codeconfigured to cause the at least one processor to delete the one or moreevent segments from the first event processing buffer; and thirdappending code configured to cause the at least one processor to appendthe one or more new event segments to the first event processing bufferand the second event processing buffer.
 18. A non-transitory computerreadable medium storing a program causing a computer to execute aprocess, the process comprising: obtaining media data; generating, fromthe media data, one or more event segments; appending the one or moreevent segments to a first event processing buffer, the one or more eventsegments comprising an event start offset for each of the one or moreevent segments based on a time at which the each of the one or moreevent segments is appended to the first event processing buffer;appending, the one or more event segments, to a second event processingbuffer, the one or more event segments comprising event dispatchinformation for the each of the one or more event segments; anddispatching the one or more event segments based on the first eventprocessing buffer and the event dispatch information in the second eventprocessing buffer; generating one or more new event segments; splittingthe one or more event segments in the first event processing buffer andthe second event processing buffer based on event duration of one ormore new event segments; and overwriting the one or more event segmentsin the first event processing buffer and the second event processingbuffer based on the event duration of the one or more new eventsegments.